System for optical recognition, interpretation, and digitization of human readable instruments, annunciators, and controls

ABSTRACT

A system for optically recognizing, interpreting, and digitizing human readable instruments, annunciators, and controls includes an image acquisition sensor operable to capture images of at least one of the instruments, annunciators, and controls; and a logic processing unit operable to decode the images captured by the image acquisition sensor and interpret the decoded images to determine a state of the at least one of the instruments, annunciators, and controls.

TECHNICAL FIELD

The present invention relates in general to optical recognition,interpretation, and digitization of human readable instruments,annunciators, and controls.

DESCRIPTION OF THE PRIOR ART

Instrumentation systems use electrical/electronic analog to digitalsystems to sense the physical state of specified electromechanicalsystems, digitize the sensor data, and store the digitized data forsubsequent analysis. Such systems, however, are costly to produce,operate and maintain. Moreover, such systems undesirably increase theweight of weight-sensitive systems.

There are ways of sensing the physical state of electromechanicalsystems well known in the art; however, considerable shortcomingsremain.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. However, the invention itself, as well asa preferred mode of use, and further objectives and advantages thereof,will best be understood by reference to the following detaileddescription when read in conjunction with the accompanying drawings, inwhich the leftmost significant digit(s) in the reference numeralsdenote(s) the first figure in which the respective reference numeralsappear, wherein:

FIG. 1 is a view of an illustrative embodiment of a rotorcraftinstrument panel;

FIG. 2 is an enlarged view of an illustrative embodiment of anannunciator panel of the instrument panel of FIG. 1;

FIG. 3 is an enlarged view of an illustrative embodiment of a torqueindicator of the instrument panel of FIG. 1;

FIG. 4 is an enlarged view of an illustrative embodiment of an airspeedindicator of the instrument panel of FIG. 1;

FIG. 5 is an enlarged view of an illustrative embodiment of aninstrument of the instrument panel of FIG. 1 having two separatesegmented light-emitting diode gauges;

FIG. 6 is an enlarged view of an illustrative embodiment of aninstrument of the instrument panel of FIG. 1 having two separate needlegauges;

FIG. 7 is an enlarged view of an illustrative embodiment of an attitudeindicator of the instrument panel of FIG. 1;

FIG. 8 is an enlarged view of an illustrative embodiment of a horizontalsituation indicator of the instrument panel of FIG. 1;

FIG. 9 is a perspective view of an illustrative embodiment of arotorcraft instrument panel and cockpit;

FIG. 10 is a graphical representation of an exemplary embodiment of asystem for optical recognition, interpretation, and digitization ofhuman readable instruments, annunciators, and/or controls;

FIG. 11 is a block diagram of an illustrative process for controllingthe overall optical recognition, interpretation, and digitizationprocess of the system of FIG. 10;

FIG. 12 is a block diagram depicting an illustrative process ofdetermining a value of an instrument or indicator as it would beinterpreted by a human operator;

FIG. 13 is a block diagram depicting an illustrative image registrationprocess whereby the actual position and orientation of an image relativeto an expected position and orientation is calculated;

FIG. 14 is a block diagram of an illustrative target segmentationprocess which determines a state of an instrument by comparing opticalproperties of a target's foreground and background images; and

FIG. 15 is a block diagram of an illustrative pattern matching techniqueused for determining control states or instrument readings.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof have been shown by wayof example in the drawings and are herein described in detail. It shouldbe understood, however, that the description herein of specificembodiments is not intended to limit the invention to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the invention as defined by the appended claims.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Illustrative embodiments of the invention are described below. In theinterest of clarity, not all features of an actual implementation aredescribed in this specification. It will of course be appreciated thatin the development of any such actual embodiment, numerousimplementation-specific decisions must be made to achieve thedeveloper's specific goals, such as compliance with system-related andbusiness-related constraints, which will vary from one implementation toanother. Moreover, it will be appreciated that such a development effortmight be complex and time-consuming but would nevertheless be a routineundertaking for those of ordinary skill in the art having the benefit ofthis disclosure.

In the specification, reference may be made to the spatial relationshipsbetween various components and to the spatial orientation of variousaspects of components as the devices are depicted in the attacheddrawings. However, as will be recognized by those skilled in the artafter a complete reading of the present application, the devices,members, apparatuses, etc. described herein may be positioned in anydesired orientation. Thus, the use of terms such as “above,” “below,”“upper,” “lower,” or other like terms to describe a spatial relationshipbetween various components or to describe the spatial orientation ofaspects of such components should be understood to describe a relativerelationship between the components or a spatial orientation of aspectsof such components, respectively, as the device described herein may beoriented in any desired direction.

The present invention represents a system for optical recognition,interpretation, and digitization of human-readable instruments,annunciators, controls, and/or the like.

For the purposes of this disclosure, the term “instrument panel” meansan apparatus including one or more instruments, annunciators, controls,and/or the like. Collectively, instruments, annunciators, controlsand/or the like are “objects” of the instrument panel. Such objects,however, are not limited to instrument panel objects but includeinstruments, annunciators, controls, and/or the like of any suitableequipment. For example, an instrument may take on the form of a gaugehaving any suitable shape or size and having a mechanical indicator orneedle that moves across the face of a fixed background to indicate thevalue of a parameter or condition under measurement. In another example,an instrument may include one or more selectively-illuminated segmentsthat display the value of a parameter or condition under measurement. Anexemplary annunciator may include one or more selectively-illuminatedsegments, such that the illumination status, e.g., on or off, bright ordim, color, etc., of one or more of the segments corresponds to thevalue of a parameter or condition under measurement. A control, forexample, refers to any apparatus that may be used to alter the state ofthe control itself and/or another apparatus. Examples of such controlsinclude, but are not limited to, switches, knobs, buttons, levers,pedals, wheels, actuators, or other such mechanisms that may bemanipulated by either a human operator or another apparatus.

As used herein, the terms “camera” and “image acquisition sensor” referto an apparatus that may be used to produce a digitized,computer-readable representation of a visual scene. Examples of suchcameras or image acquisition sensors include, but are not limited to,digital cameras, optical scanners, radar systems, ultrasonic scanners,thermal scanners, electronic scanners, and profilometric devices.

For the purposes of this disclosure, the term “foreground” means imageelements of interest during interpretation of a visual target. Forexample, a needle of a gauge may be considered as foreground in certainoperations. As used herein, the term “background” means image elementsthat are of little or no interest during the interpretation of a visualtarget. For example, markings on a dial face of a gauge may beconsidered as background in certain operations.

It should be noted that the system and method of the present inventionmay be used in many diverse implementations, as is discussed in greaterdetail herein. The system is described in detail herein in relation to arotorcraft instrument panel, although the scope of the present inventionis not so limited. Rather, the system and method of the presentinvention may be used in relation to any human-readable instrument orinstruments, irrespective of the type of equipment with which theinstrument or instruments are associated.

FIG. 1 depicts a rotorcraft instrument panel 101 including one or moreinstruments, annunciators, and the like, of various styles. In theillustrated embodiment, an annunciator panel 103, also shown in FIG. 2,includes one or more discrete annunciator segments, such as anannunciator segment 105, with each segment acting as an indicator for aspecific, discrete condition. Examples of conditions indicated by suchannunciator segments include, but are not limited to, engine fire, lowfuel, battery hot, generator failure, and the like.

As discussed above, rotorcraft instrument panel 101 includes one or moregauges, such as a gauge 107. Referring now to FIG. 3, gauge 107 includesa generally circular, graduated indicator or dial 301 on its face in theillustrated embodiment. The particular embodiment of gauge 107illustrated in FIG. 3 employs segmented light-emitting diodes 303 thatare illuminated to indicate the value of the parameter or conditionbeing measured. Note that, in this case, the values on dial 301 arenon-linear, i.e., a given angular displacement has different valuesdepending upon which region of dial 303 the angular displacement isimposed. For example, the angular displacement in a region betweenvalues of 0 and 4, generally at 305, is about the same as the angulardisplacement in a region between values of 5 and 7, generally at 307. Inother words, the gauge has about twice the resolution between values of5 and 7 as the resolution between values of 0 and 4. The particularembodiment of gauge 107 provides an indication of engine torque. Gauge107 further includes a five-character alphanumeric display 309 fordisplaying a numeric representation of a condition or parameter beingmeasured or system status information during diagnostic procedures. Inthe illustrated embodiment, display 309 provides a numeric readout ofengine torque or system status information during diagnostic procedures.

Referring to FIGS. 1 and 4, the illustrated embodiment of rotorcraftinstrument panel 101 further includes a gauge 109 having a circular,graduated indicator or dial 401 on its face. Gauge 109 employs a needle403 to indicate a value of a condition or parameter being measured.

Referring now to FIGS. 1 and 5, the illustrated embodiment of rotorcraftinstrument panel 101 further includes an instrument 111 having twoseparate segmented light-emitting diode gauges 501 and 503. In theillustrated embodiment, gauge 501 indicates fuel pressure in pounds persquare inch and gauge 503 indicates generator output in amperes.

Referring to FIGS. 1 and 6, the illustrated embodiment of rotorcraftinstrument panel 101 further includes an instrument 113 having twoseparate needle gauges 601 and 603. Gauge 601 employs a needle 605 andgauge 603 includes a needle 607. Needles 605 and 607 move independently.In the illustrated embodiment, needle 605 indicates main rotorrotational speed as a percentage of normal rotor rotational speed forflight operations in relation to an outer scale 609. Needle 607indicates engine turbine rotational speed as a percentage of normalengine turbine speed for flight operations in relation to an inner scale611.

Referring now to FIGS. 1 and 7, the illustrated embodiment of rotorcraftinstrument panel 101 further includes an instrument 115 that combinestwo graduated scale gauges 701 and 703 and a discrete annunciator 705into one instrument. The particular instrument 115 depicted in FIGS. 1and 7 is known as an “attitude indicator.” In the illustratedembodiment, the bank angle or roll attitude of the rotorcraft isindicated using gauge 701 by motion of an outer dial 707 with respect toa fixed pointer 709. The measured value is read from a mark on outerdial 707 corresponding with pointer 709. The region in the center ofinstrument, i.e., gauge 703, uses a graduated scale 711 on a movablecard 713 to indicate rotorcraft pitch attitude corresponding to a fixedpointer 715. Annunciator 705, known as a “barber pole,” is a discreteannunciator indicating that instrument 115 is invalid or non-operationalwhenever annunciator 705 is visible to the human eye.

Still referring to FIG. 7, instrument 115 further includes a control inthe form of a knob 719 that allows arbitrary setting of a zero pitchattitude reference. Instrument 115 also includes a control in the formof a knob 721 to cage gyroscopes operably associated with instrument115.

Referring now to FIGS. 1 and 8, the illustrated embodiment of rotorcraftinstrument panel 101 further includes an instrument 117 that combines aplurality of gauges, markers, scales, pointers, and the like, along witha plurality of discrete annunciators. In the illustrated embodiment,instrument 117 takes on the form of a horizontal situation indicator.The aircraft heading is displayed on a rotating azimuth or compass card801 under a lubber line 803. A course deviation bar 805 operates with afixed navigational reference receiver, such as a very high-frequencyomni-directional range/localizer (VOR/LOC) navigation receiver, a veryhigh-frequency omni-directional range tactical air navigation (VORTAC)receiver, or an instrument landing system (ILS) receiver to indicateeither left or right deviations from the course that is selected with acourse select pointer 807. Course deviation bar 805 moves left or rightto indicate deviation from a center of scale 809. The desired course isselected by rotating course select pointer 807 with respect to compasscard 801 by means of a course set knob 811. A fixed aircraft symbol 813and course deviation bar 805 display the aircraft relative to theselected course as though the pilot was above the aircraft looking down.A glide slope deviation pointer, which is not shown in FIG. 8 as thepointer is only visible when a valid glide slope signal is beingreceived, indicates the relationship of the aircraft to the glide slope.When the glide slope deviation pointer is above the center position of ascale 815, the aircraft is above the glide slope and an increased rateof descent is required.

Still referring to FIGS. 1 and 8, annunciator 817 indicates, whenvisible to the human eye, that the navigation functions of instrument117 are inoperative. Annunciator 819 indicates, when visible to thehuman eye, that the heading functions of instrument 117 are inoperative.Annunciator 821 indicates a heading reference set by an operator of therotorcraft using a control knob 823 of instrument 117.

Referring to FIG. 1, the illustrated embodiment of instrument panel 101further includes a guarded toggle switch 119, which is a control to openor close a fuel valve of the rotorcraft. Instrument panel 101 furtherincludes a push button switch 121 that is used to control an operationalmode, i.e., automatic or manual, of a full authority digital enginecontrol (FADEC). A face of button 121 illuminates as an annunciator ofthe currently selected FACEC mode.

FIG. 1 also shows numerous other instruments, annunciators, and controlspresent, visible to the human eye, and available to a crew duringrotorcraft operations. From a single image represented in FIG. 1, thesystem described herein recognizes, interprets, and produces digitaltime histories for a plurality of annunciators, instruments, andcontrols using a single image acquisition sensor rather than sensorscorresponding to each of the annunciators, instruments, and controls. Inthe implementation illustrated in FIG. 1, the system described hereinrecognizes, interprets, and produces digital time histories for 57annunciators, 26 instruments, and seven controls using a single imageacquisition sensor rather than 90 independent sensors required usingconventional instrumentation. Accordingly, the cost, weight, andcomplexity of the system described herein are significantly less thanfor conventional sensing systems.

The examples described in the preceding paragraphs are characteristic ofthe types of a devices and apparatus to which the invention describedherein can be applied; however, neither the listed items nor thehelicopter context are in any way exhaustive as to the opportunities forapplication of this invention. For example, the present invention may beused with or incorporated into various types of equipment, systems,devices, and the like other than with or in rotorcraft.

FIG. 9 is a perspective view of a portion of a cockpit of a rotorcraftas captured by a digital cockpit video recorder (not shown in FIG. 9),i.e., an image acquisition sensor. Visible in FIG. 9 is a collectivehead 901 disposed at a free end of a collective lever (not shown in FIG.9). Collective head 901 is used to control a thrust magnitude of a mainrotor of the rotorcraft by varying an incidence angle of the main rotorblades of the rotorcraft. In practical use, raising collective head 901generally increases thrust, while lowering collective head 901 generallydecreases thrust. Also, visible in FIG. 9 is a cyclic grip 903. Movementof cyclic grip 903 causes cyclical pitch variations of the main rotorblades of the rotorcraft, resulting in redirection of main rotor thrust,to control direction of flight. Anti-torque or tail rotor pedals,indicated collectively as 905, are also visible in FIG. 9. Movement ofpedals 905 causes a change in tail rotor thrust of the rotorcraft,resulting in a change in yaw attitude. Collective head 901, cyclic grip903, and tail rotor pedals 905 are further examples of controls havingpositions that may be processed by the system described herein.

According to the system described herein, a digital representation of animage of the instruments, annunciators, and/or controls is interpretedusing computerized means in a way corresponding to interpretation byhuman observation. In the case of annunciators, the state of theannunciator defines a quality or state being measured. Changes inchromaticity and/or intensity between annunciated states can bedetermined by the present system. In the case of various types ofgauges, every possible discrete value for a gauge maps to a discreteposition on a digitized image of that gauge. By using the present systemto determine the position of the variable features of a gauge, e.g., aneedle, the value indicated by the gauge at the time of image capturecan be determined. Regarding a needle gauge, for example, if analysis ofa digitized image determines that the needle is located at a position ona digitized image that maps to 100 knots of airspeed, the system canstore 100 knots as the airspeed value at the time of image capture.Thus, the invention described herein can be used in place of moreexpensive, more complex, and bulkier conventional wired or even wirelessinstrumentation systems.

FIG. 10 depicts an exemplary embodiment of a system 1001 for opticalrecognition, interpretation, and digitization of human-readableinstruments, annunciators, and controls. The objects labeled as 1003represent objects to be recognized, interpreted, and digitized. Forexample, in a rotorcraft implementation, objects 1003 represent objectsin a rotorcraft cockpit, such as the instruments, annunciators, andcontrols described herein. System 1001 comprises an image acquisitionsensor 1005, such as a digital camera that captures images of arotorcraft instrument panel and the surrounding environment. In theillustrated embodiment, image acquisition sensor 1005 combines the imagecapture and encoding steps in one device, although the scope of thepresent invention is not so limited. Rather, in an alternativeembodiment, image acquisition sensor 1005 captures the image and anotherdevice is used to encode the image into a digital representation. System1001 further comprises a logic processing unit 1007, which, in theillustrated embodiment, is a digital computer executing software todecode and interpret the digitally-encoded image. Logic processing unit1007 stores the states, preferably as time-based states, of one or moreinstruments, annunciators, and controls of interest represented in theimage to a data storage device 1009. In some embodiments, logicprocessing unit 1007 creates pseudo-parameters to augment informationavailable to the crew of the rotorcraft during flight. Pseudo-parametersare data items produced by combining two or more other parametersmathematically, lexically, and/or logically. For example, a verticalspeed pseudo-parameter could be produced by taking the time derivativeof altitude. Wind velocity, speed, and direction could be derived fromsatellite-based global positioning system data, airspeed, and heading. Adownwind approach to land could be inferred from vertical speed, e.g.,sink rate, low airspeed, and wind direction, while at low altitude. Insome embodiments, system 1001 further includes a ground data processor1011, e.g., a general-purpose computer using custom software. It shouldbe noted that the components of system 1001 depicted in the drawingsand/or the functions performed by the components may be combined,separated, and/or redistributed depending upon the particularimplementation of system 1001.

FIG. 11 depicts a block diagram representing an illustrative embodimentof a method for optically recognizing, interpreting, and digitizinghuman-readable instruments, annunciators, and controls. In oneimplementation, the method is embodied in software encoded in media thatcan be read and executed by computing device. Upon initialization of themethod (block 1101), a configuration file is read to provide adefinition of the image to be interpreted (block 1103). The filecontains information including, but not limited to the number, type, andlocation of all objects of interest, i.e., instruments, annunciators,and/or controls, to be interpreted; a definition of a scan region foreach object; value maps associated with discrete locations within thescan region in the digitized image for use with threshold crossingtechniques, as is described in greater detail herein; a mathematicalrepresentation of a reference region of the image for use in imageregistration; a mathematical representation of reference patterns usedto determine states for objects interpreted using pattern matchingtechniques; chromaticity and intensity information for use ininterpreting illumination status for lighted annunciators; chromaticityand intensity information for use in interpreting background versusforeground information; and/or other parameters needed by the method toensure efficient operation, such as startup parameters, preferences, andthe like.

It should be noted that the configuration file stores information forthe processing of images to eliminate redundant processing.Human-readable instruments, annunciators, and control can be recognizedwithout the configuration file and the scope of the present inventionencompasses such an embodiment. In such an embodiment, the registrationprocess is omitted. The embodiment is particularly useful inimplementations wherein computer processing capacity is not an issue inthe timely providing of real-time analysis.

In certain embodiments, system 1001 includes one or more computerprograms encoded in media that can be read and executed by a computingsystem to facilitate the creation and maintenance of the configurationfile using both automated and user-interactive methods. In suchembodiments, previously defined objects can be automatically recognizedusing pattern matching techniques and manual techniques where anoperator can define the configuration using custom tools in a graphicaluser interface (GUI).

Still referring to FIG. 11, the method assumes a piecewise continuousstream of images for analysis and opens an output file (block 1105),such as in data storage device 1009 of FIG. 10, to which a series ofvalues as interpreted for each subsequent image can be written. Theoutput file effectively contains a time history of all parametersinterpreted and analyzed according to the definitions read from theconfiguration file.

The method enters a conditional loop 1107 and tests for the presence ofa digital image representation to be interpreted (block 1109). If theimage is present, the program decodes the image (block 1111) asnecessary to create an indexable representation of the image. Thedecoding process may produce the indexable representation in any one ormore possible forms according to the characteristics of the image beingprocessed. In one embodiment, the image is a two-dimensional pixelbitmap or raster stored (block 1113) as two-dimensional array incomputer memory. System 1001, however, is equally applicable tothree-dimensional images. Gauges are then read from the raster (block1115), as is discussed in greater detail with regard to FIG. 12. Thevalue of each gauge is then written to the output file (block 1117). Ifthe image is not present (block 1109), stored variables and the like arecleaned up and the method ends.

FIG. 12 depicts one particular embodiment of reading the gauges from theraster (block 1115) in FIG. 11. While the method depicted in FIG. 12 isdescribed regarding two-dimensional techniques, the scope of the presentinvention is not so limited. The first step is to “register” the image(block 1201) to ensure proper alignment between the coordinate system ofcurrent raster and the coordinate system used in the creation of theconfiguration file. FIG. 13 shows that the first step in theregistration process (block 1201) is to calculate a Fourier transform ofthe current rasterized image (block 1301). In this embodiment, fastFourier transform (FFT) techniques are used. The reference region readfrom the configuration file is a conjugate of an FFT of a referenceimage. The next step is to element-wise multiply the reference FFTconjugate with the FFT of the current image (block 1303). The resultingproduct is then normalized element-wise such that all FFT values rangebetween −1 and 1 (block 1305). The normalization process is notabsolutely necessary but aids in overcoming errors due to imagevariations such as differences in lighting and/or optical noise. Thenext step is to compute the inverse FFT (IFFT) of the normalized product(block 1307). The final step is to find the row and column indices ofthe maximum value in the array from the IFFT (block 1309). The processis then repeated using polar coordinates to correct rotationaldifferences. The resultant row and column indices are applied as offsetsto row and column indices read from the configuration file (block 1311)and the process returns to the next step in FIG. 12 (block 1313).

It should be noted that the process of registration can be accomplishedin other ways, which are encompassed within the scope of the presentinvention. For example, the region to be registered does not have to bea rectangular array. A single row of the raster can be used to obtainhorizontal or lateral image shifts and a single column of the raster canbe used to obtain vertical shifts. While the use of a single row andcolumn of the raster may fail to determine the image shift when theshift involves both vertical and horizontal translation, the use of asingle row and column of the raster may be sufficient or even desirablein certain implementations. Other ways of registering the image arecontemplated by the present invention. Regarding embodiments of thepresent invention wherein registration is used, the manner in whichregistration is accomplished is immaterial to the practice of theinvention.

The overall outcome of the registration process may be referred to as“image stabilization,” inasmuch as image shifts due to changes in therelative position and orientation between the image acquisition sensorand the instrument panel in this case can be corrected for translationaland/or rotational errors.

Referring back to FIG. 12, once registration is complete, thecoordinates of each target region defined by the configuration file areadjusted by adding the row and column offsets from the registrationprocess (block 1203). The next step is to enter a loop 1205 iteratingthrough each threshold crossing type object defined in the configurationfile. Once inside the loop, each pixel in the target region of theregistered, current image is scanned (block 1207). A median filter isthen applied to each pixel (1209). The value produced by the medianfilter is then assigned to the pixel. The median filter is used toreduce the effects of noise in the acquired image.

FIG. 14 presents the process flow for segmenting the target region intoforeground and background (block 1211 in FIG. 12). The first step is tocompute a threshold value (block 1401) for the entire target regionusing the results of the median filter step. The present inventioncontemplates many ways for determining a threshold value, but a midpointcalculation between the minimum and maximum values for the target regionhas been tested and shown to be acceptable.

Threshold crossing detection is one method of normalizing the image toaccount for lighting variations. The present method is capable ofoperating under a variety of lighting conditions. In some cases objectsare illuminated by ambient light from sun, moon, or area lighting. Insome cases objects are illuminated by artificial backlighting ordirected face lights. The color and intensity will vary widely. Forexample, night flying generally uses low intensity red artificiallighting. Other techniques including but not limited to grayscaleanalysis and/or negative image analysis are also options for handlingvariation in chromaticity and/or intensity of light.

Once the threshold value has been established, the intensity of eachpixel in the target is retrieved (block 1403) and the intensity of eachpixel is compared to the threshold (block 1405) and classified as eitherforeground (block 1407) or background (block 1409). Once all pixels havebeen scanned (block 1411), the process returns (block 1413) to the nextstep in FIG. 12.

Referring back to FIG. 12, the next step is to pick the most likelycandidate from among the foreground pixels which correspond to thelocation of the instrument's variable feature (block 1213). The methodsfor determining the most likely candidate can be as varied as the typesof objects being interpreted. For example, a needle position might bedetermined by calculating the centroid of a cluster of foregroundpixels. A segmented LED gauge value might be interpreted by determiningthe location at which there is an abrupt and non-reversing change fromforeground to background pixels in the target region. An annunciatorstate might by interpreted by detecting the presence of one or moreforeground pixels in the target region, with the number dependent uponsuch factors as quality of the image, desired confidence level, or othermeasures.

FIG. 15 refers to image interpretations using pattern matchingtechniques, which is one embodiment of block 1213 in FIG. 12. Thetechniques described herein are essentially a repeat of the imageregistration process using a stored mathematical representation of areference pattern to determine the state of an object (block 1501). Theimage raster is windowed around the control to obtain a smaller raster(block 1503) and the registration process is conducted over the smallerregion of the image (block 1505). The phase correlation result isstatistically analyzed to determine the probability of match of thereference pattern (block 1507) and the process returns (block 1509) tothe next step in FIG. 12. Numerous other pattern matching variationswell known in the literature could likewise be employed. Patternmatching techniques might be a preferred method for determining statesof particular knobs, switches, and/or levers not well suited forthreshold crossing techniques.

Threshold crossing and pattern matching techniques can also be used,either singly or in combination, to interpret the position of primaryflight controls such as the collective to which collective head 901 isattached, cyclic grip 903, or tail rotor pedals 905, all shown in FIG.9.

Referring again to FIG. 12, once the values and/or states of allinstruments, annunciators, and/or controls have been interpreted for thecurrent image (block 1215), the values are stored in an output file(block 1117 of FIG. 11), preferably stored on removable media. Bothcrashworthy and non-crashworthy removable media implementations arecontemplated by the present invention. The entire process is thenrepeated for the next image.

Various methods for data quality evaluations are possible using thisinvention. Threshold crossing detection is an effective method fordetecting when an object is occluded from the camera view, such as by apilot's arm as he reaches in front of the instrument panel to press abutton or turn a knob. The minimum, maximum, and threshold valuescalculated from a median filtered image of a bare arm or a shirt sleevewill be very close, allowing the system to interpret that the targetregion is occluded. Redundant measurements are also used assess thequality of the interpreted value. For example, the roll angle from theattitude indicator 115 (shown in FIGS. 1 and 7) can be interpreted fromtarget regions scanned on both the left and right sides of the rotatingouter dial 707. In addition to multiple target regions,multi-directional scans provide redundancy. For example scanningleft-to-right followed by a right to left scan may detect blurriness ormultiple threshold crossings in which case the image may be “de-blurred”using Fourier optics techniques. If multiple threshold crossings aredetected, various expert and/or statistical wild point editingtechniques may be employed. Interpretations made using pattern matchingtechniques can check for multiple states, up/down, on/off, etc. Datafrom one source can also be compared and correlated with data from othersources, including other optically interpreted sources to assess dataquality. Suspect data can be tagged by such means as sentinel values ordata quality bits.

In some cases, data interpreted from one source may be used inconjunction with data from other sources (optical or otherwise) forvarious purposes. One such case would be estimating wind velocity anddirection using optically interpreted airspeed and heading data inconjunction with ground speed and ground track data from a wired GlobalPositioning System (GPS) sensor. Wind speed information thus derivedcould be stored as a time history “pseudo-item” as well as displayed tothe pilot real-time in flight.

The present invention provides significant advantages, including: (1)providing a cost-effective means for monitoring human-readableinstruments, annunciators, and controls in an aircraft, such as arotorcraft; (2) providing a cost-effective means for monitoringhuman-readable instruments, annunciators, and controls associated withany type of equipment or machinery, such as industrial processingmachinery, material handling equipment, machine tools, or the like; (3)providing a lower complexity means for monitoring the state of aircraftsystems, equipment, machinery, or the like; and (4) providing a lowerweight means for monitoring the state of aircraft systems, equipment,machinery, or the like.

The particular embodiments disclosed above are illustrative only, as theinvention may be modified and practiced in different but equivalentmanners apparent to those skilled in the art having the benefit of theteachings herein. Furthermore, no limitations are intended to thedetails of construction or design herein shown, other than as describedin the claims below. It is therefore evident that the particularembodiments disclosed above may be altered or modified and all suchvariations are considered within the scope and spirit of the invention.Accordingly, the protection sought herein is as set forth in the claimsbelow. It is apparent that an invention with significant advantages hasbeen described and illustrated. Although the present invention is shownin a limited number of forms, it is not limited to just these forms, butis amenable to various changes and modifications without departing fromthe spirit thereof.

The invention claimed is:
 1. A system for optically recognizing,interpreting, and digitizing human readable instruments, annunciators,and controls, comprising: an image acquisition sensor operable tocapture images of multiple instruments, annunciators, and controls; anda logic processing unit operable to decode the images captured by theimage acquisition sensor and interpret the decoded images to determine astate of the multiple instruments, annunciators, and controls; whereinthe human readable instruments, annunciators, and controls are objectsin an aircraft cockpit; wherein the system detects when at least one ofthe objects is occluded from the images captured by the imageacquisition sensor; wherein the system groups the objects into anoccluded group of objects and a non-occluded group of objects; andwherein the system continues to determine the state of the non-occludedgroup of objects while the at least one of the multiple instruments,annunciators, and controls is occluded.
 2. The system of claim 1,wherein the logic processing unit is operable to store the state of atleast one of the multiple instruments, annunciators, and controls. 3.The system of claim 1, wherein the logic processing unit is operable tostore time-based states of at least one of the multiple instruments,annunciators, and controls.
 4. The system of claim 1, wherein the logicprocessing unit creates a pseudo-parameter from the state of at leastone of the multiple instruments, annunciators, and controls.
 5. Thesystem of claim 1, further comprising: a ground data processor operablyassociated with the logic processing unit.
 6. The system of claim 1,wherein the aircraft is a rotorcraft.
 7. A method for opticallyrecognizing, interpreting, and digitizing human readable instruments,annunciators, and controls, comprising: reading a configuration file toprovide a definition of a target region of an image to be interpreted ofat least one of the instruments, annunciators, and controls to beinterpreted; opening an output file for storing the image to beinterpreted; determining whether a captured image to be interpretedexists; if the captured image exists: decoding the captured image;determining a state of the at least one of the instruments,annunciators, and controls based upon the decoded captured image and thedefinition in the configuration file; detecting when the at least one ofthe instruments, annunciators, and controls are not occluded from thecaptured image; ignoring the state of the at least one of theinstruments, annunciators, and controls that are occluded; and writingthe state of the at least one of the non-occluded instruments,annunciators, and controls to the output file.
 8. The method of claim 7,wherein the configuration file includes one or more of: a number of theat least one of the instruments, annunciators, and controls to beinterpreted; types of the at least one of the instruments, annunciators,and controls to be interpreted; locations of the at least one of theinstruments, annunciators, and controls to be interpreted; a definitionof a scan region of the at least one of the instruments, annunciators,and controls to be interpreted; value maps associated with the locationsof the at least one of the instruments, annunciators, and controls to beinterpreted; a mathematical representation of a reference region of theimage for use in image registration; a mathematical representation ofreference patterns used to determine states of the at least one of theinstruments, annunciators, and controls to be interpreted; chromacityand intensity information for use in interpreting illumination statusfor an annunciator; chromacity and intensity information for use ininterpreting background versus foreground information of the image; andstartup parameters and preferences.
 9. The method of claim 7, whereinthe output file includes a time history of parameters interpreted andanalyzed based upon definitions read from the configuration file. 10.The method of claim 7, wherein decoding the image results in atwo-dimensional pixel bitmap or raster stored as a two-dimensionalarray.
 11. The method of claim 7, wherein determining the state of theat least one of the instruments, annunciators, and controls isaccomplished by: registering the captured image with the definition ofthe target region of the image to be interpreted; adjusting coordinatesof the target region defined by the configuration file based uponregistering the captured image; scanning each pixel of the registered,captured image; applying a median filter to each pixel of theregistered, captured image; segmenting the target region into foregroundand background; and picking foreground pixels corresponding to avariable feature of the at least one of the instruments, annunciators,and controls to be interpreted.
 12. The method of claim 11, whereinregistering the captured image is accomplished by: computing a Fouriertransform of the captured image; element-wise multiplying a referenceFourier transform conjugate from the configuration file with the Fouriertransform of the captured image; computing an inverse Fourier transformof the multiplied reference Fourier transform conjugate and the Fouriertransform of the captured image; finding row and column indices of amaximum value in an array resulting from the inverse Fourier transform;and storing the row and column indices.
 13. The method of claim 12,further comprising: element-wise normalizing the multiplied referenceFourier transform conjugate and the Fourier transform of the capturedimage prior to computing the inverse Fourier transform.
 14. The methodof claim 11, wherein segmenting the target region into foreground andbackground is accomplished by: computing a threshold value for thetarget region based upon results of applying the median filter to eachpixel of the registered, captured image; retrieving an intensity of eachpixel in the target area of the captured image; comparing the intensityof each pixel to the threshold value; and classifying each pixel asforeground or background.
 15. The method of claim 11, wherein pickingforeground pixels corresponding to a variable feature of the at leastone of the instruments, annunciators, and controls to be interpreted isaccomplished by: windowing the captured image around the target area;registering the windowed area of the captured image with the definitionof the target region of the image to be interpreted; and determining thestate of the at least one of the instruments, annunciators, and controlsto be interpreted based on the result of registering the windowed area.